A general-purpose sentence-level nonsense detector

I have constructed a sentence-level nonsense detector, with the goal of discriminating well-formed English sentences from the large volume of fragments, headlines, incoherent drivel, and meaningless snippets present in internet text. For many NLP tasks, the availability of large volumes of internet text is enormously helpful in combating the sparsity problem inherent in modeling language. However, the derived models can be easily polluted by ungrammatical text and spam, and automated means are necessary to filter this and provide high-quality training data. There is scarce precedent in the literature for a direct nonsense-detection system, but similar problems exist in the context of spam filtering and computational sentence completion. For spam filtering, recent research has focused on the problem of “Bayesian poisoning” (Hayes, 2007), whereby Naive Bayes-based filters are defeated by the inclusion of random words or text snippets that make the spam vocabulary appear similar to normal text. Solutions to this problem propose combining multiple sources of information, such as higher n-grams, in order to introduce greater context (Upasana and Chakravarty, 2010). Sentence completion is an example of a standard NLP task where the system must consider a variety of possible solutions and discriminate between “sensical” and “nonsensical” answers. While generally operating on a more restricted domain (e.g. SAT questions), the fundamental goal is similar enough that similar techniques can be applied. The Microsoft Research Sentence Completion Challenge (Zweig and Burges, 2011) highlights several approaches to this task, including neural networks and dependency parsing but also showing strong performance with lighterweight n-gram and distributional models (Zweig et al., 2012). I take a lightweight approach, using a mix of heuristics, lightweight token-based features, part-of-speech features, and language model scores as features to avoid computationally-intensive parsing or neural networks. I structure my system as a binary classification task on a heterogeneous feature space, consisting of to produce a final answer of “sentence” or “nonsense” for a given line of text. I implement this project in a mix of Java and Python, using Java to interface with Stanford CoreNLP for feature extraction, and Python (with the excellent pandas and scikit-learn libraries) for data management, classifier implementation, and analysis.